Papers with Artificial General Intelligence
i-Code Studio: A Configurable and Composable Framework for Integrative AI (2024.emnlp-demo)
Copied to clipboard
Yuwei Fang, Mahmoud Khademi, Chenguang Zhu, Ziyi Yang, Reid Pryzant, Yichong Xu, Yao Qian, Takuya Yoshioka, Lu Yuan, Michael Zeng, Xuedong Huang
| Challenge: | Existing frameworks for Integrative AI lack flexibility and composability to handle multimodal tasks. |
| Approach: | They propose a configurable framework for Integrative AI that orchestrates multiple pre-trained models to conduct complex multimodal tasks. |
| Outcome: | The proposed framework achieves impressive results on zero-shot multimodal tasks . it can communicate and personalize for users, and it can be used in a multimodal agent . |
COGEN: Abductive Commonsense Language Generation (2023.acl-short)
Copied to clipboard
| Challenge: | Existing training methods for NLP models to perform on two main tasks are needed to introduce these capabilities into the field of reasoning. |
| Approach: | They propose a model that integrates commonsense reasoning with contextual filtering to improve the inference. |
| Outcome: | The proposed model outperforms existing models and sets new state-of-the-art in regards to alphaNLI and alphaNGG tasks. |
LLMs for Mathematical Modeling: Towards Bridging the Gap between Natural and Mathematical Languages (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have demonstrated strong performance across various natural language processing tasks, but their proficiency in mathematical reasoning remains a key challenge. |
| Approach: | They propose a process-oriented framework to evaluate LLMs' ability to construct mathematical models, using solvers to compare outputs with ground truth. |
| Outcome: | The proposed framework evaluates LLMs' ability to construct mathematical models, using solvers to compare outputs with ground truth. |
AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models (2024.findings-naacl)
Copied to clipboard
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, Nan Duan
| Challenge: | Traditional benchmarks for evaluating foundation models often fail to accurately represent their general abilities for human-centric tasks. |
| Approach: | They propose a bilingual benchmark to assess foundation models in the context of human-centric standardized exams such as college entrance exams, law school admission tests, and math competitions. |
| Outcome: | The proposed benchmark exceeds the average human performance on SAT, LSAT, and math competitions with 95% accuracy and 92.5% on the Chinese college entrance English exam. |
Safe Inputs but Unsafe Output: Benchmarking Cross-modality Safety Alignment of Large Vision-Language Models (2025.findings-naacl)
Copied to clipboard
Siyin Wang, Xingsong Ye, Qinyuan Cheng, Junwen Duan, Shimin Li, Jinlan Fu, Xipeng Qiu, Xuanjing Huang
| Challenge: | Recent studies focus on single-modality threats, but this approach fails to address cross-modal safety alignment. |
| Approach: | They propose a safety alignment challenge to evaluate cross-modality safety alignment . they propose 'Safe Inputs but Unsafe Output' to consider safety of single modalities . |
| Outcome: | The proposed safety alignment challenge examines cases where modalities are safe independently but could lead to unsafe outputs when combined. |
Humans Keep It One Hundred: an Overview of AI Journey (2020.lrec-1)
Copied to clipboard
Tatiana Shavrina, Anton Emelyanov, Alena Fenogenova, Vadim Fomin, Vladislav Mikhailov, Andrey Evlampiev, Valentin Malykh, Vladimir Larin, Alex Natekin, Aleksandr Vatulin, Peter Romov, Daniil Anastasiev, Nikolai Zinov, Andrey Chertok
| Challenge: | Artificial General Intelligence (AGI) is showing growing performance in numerous applications - beating human performance in Chess and Go, using knowledge bases and text sources to answer questions and even pass human examination. |
| Approach: | They propose to use knowledge bases and text sources to answer questions to improve AI performance on knowledge bases, reasoning and text generation. |
| Outcome: | The proposed AI Journey system passed the final native language exam in Russian with a high score of 69%, with 68% being an average human result. |
UTMath: A Benchmark for Math Evaluation with Unit Test (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Prevailing benchmarks for mathematical reasoning include MATH and AIME . predicated on single-instantiation problems with fixed numbers, these models leave generalization on isomorphic problem variants untested. |
| Approach: | They propose a mathematical reasoning benchmark that quantifies solution accuracy and solution space generality. |
| Outcome: | The proposed model solves 1,053 problems spanning 9 mathematical domains . the best-performing model solved only 32.57% of the problems . |
LongLeader: A Comprehensive Leaderboard for Large Language Models in Long-context Scenarios (2025.naacl-long)
Copied to clipboard
Pei Chen, Hongye Jin, Cheng-Che Lee, Rulin Shao, Jingfeng Yang, Mingyu Zhao, Zhaoyu Zhang, Qin Lu, Kaiwen Men, Ning Xie, Huasheng Li, Bing Yin, Han Li, Lingyun Wang
| Challenge: | LongLeader aims to assess different LLMs' long-context comprehension abilities . long-constext comprehension is a key bottleneck for many use cases . |
| Approach: | They propose a leaderboard to assess different LLMs' long-context comprehension abilities . they offer open-source access to the benchmarks and maintain a dedicated website . |
| Outcome: | The proposed model assesses different LLMs on selected benchmarks and provides open-source access to the benchmarks. |
Breaking Language Barriers: Cross-Lingual Continual Pre-Training at Scale (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have made significant strides towards Artificial General Intelligence, but training them from scratch is prohibitively expensive. |
| Approach: | They propose to continuously pre-train LLMs from existing pre-trained LLM models by using a set of parameters instead of randomly initializing them. |
| Outcome: | The proposed approach saves significant resources and accelerates convergence and performance. |
MCS-Bench: A Comprehensive Benchmark for Evaluating Multimodal Large Language Models in Chinese Classical Studies (2025.acl-long)
Copied to clipboard
| Challenge: | Multimodal Large Language Models (MLLMs) have advanced visual and language understanding, but their potential in Chinese Classical Studies (CCS) remains underexplored due to the lack of specialized benchmarks. |
| Approach: | They propose to develop a multimodal benchmark specifically designed for Chinese Classical Studies across multiple subdomains to bridge this gap. |
| Outcome: | The proposed benchmark spans seven core subdomains with a total of 45 meticulously designed tasks. |
GAOKAO-MM: A Chinese Human-Level Benchmark for Multimodal Models Evaluation (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing datasets focus on primary perception abilities and commonsense knowledge, or have low level of text comprehension difficulty, which are insufficient to reflect comprehensive capabilities of large vision-language models. |
| Approach: | They propose a multimodal benchmark based on the Chinese College Entrance Examination (GAOKAO) which sets human-level requirements for the model’s abilities, including perception, understanding, knowledge and reasoning. |
| Outcome: | The proposed model derives from native Chinese context and sets human-level requirements for its abilities, including perception, understanding, knowledge and reasoning. |
Large Language Models are biased to overestimate profoundness (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in natural language processing have been suggested to approach AI . however, it is still unclear whether LLMs possess similar reasoning abilities to humans . |
| Approach: | They evaluate GPT-4 and other LLMs in judging the profoundness of mundane statements . they find a significant correlation between the LLM and humans . |
| Outcome: | The proposed model overestimates the profoundness of nonsensical statements . the model overstates the profound nature of non-senior statements, the study finds . |
Cultivating Gaming Sense for Yourself: Making VLMs Gaming Experts (2025.acl-long)
Copied to clipboard
| Challenge: | Recent efforts leverage Vision Language Models (VLMs) as direct controllers, often pausing the game to analyze screens and plan action through language reasoning. |
| Approach: | They propose a paradigm shift in gameplay agent design that uses Vision Language Models as a developer instead of direct control. |
| Outcome: | The proposed framework achieves fluent gameplay in diverse genres, including ACT, FPS, and Flappy Bird, setting a new benchmark for game-playing agents. |
SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing multi-modal large language models typically adopt the cascade paradigm, preventing inter-modal knowledge transfer. |
| Approach: | They propose a large language model with intrinsic cross-modal conversational abilities . they construct a cross-text speech instruction dataset and employ a three-stage training strategy . |
| Outcome: | The proposed model can follow cross-modal human instructions and handle multiple modalities with one model. |
Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoning (2026.findings-acl)
Copied to clipboard
Yibo Yan, Shen Wang, Jiahao Huo, Jingheng Ye, Zhendong Chu, Xuming Hu, Philip S. Yu, Carla P Gomes, Bart Selman, Qingsong Wen
| Challenge: | Current scientific reasoning models struggle with generalization across domains and fall short of multimodal perception. |
| Approach: | They propose to use multimodal large language models to integrate text, images, and other modalities to enhance scientific reasoning. |
| Outcome: | The proposed models can integrate text, images, and other modalities and improve reasoning across disciplines. |